Skip to content

benchmark(opus-5-5): Opus 5.5 on Bedrock via strands and omp, v2 dataset - #195

Merged
aarora79 merged 2 commits into
aarora79:mainfrom
shekharprateek:feat/strands-harness
Oct 4, 2026
Merged

aarora79 merged 2 commits into
aarora79:mainfrom
shekharprateek:feat/strands-harness

Conversation

@shekharprateek

@shekharprateek shekharprateek commented Oct 2, 2026 •

Copy link
Copy Markdown

Summary

  • Benchmark results for Claude Opus 5.5 on Amazon Bedrock using two harnesses on the v2 dataset (21 tasks, complexity-balanced):
    • Strands Agents SDK (--agent strands): mean 81.21, mean cost $2.46/task
    • omp (--agent omp): mean 81.16, mean cost $4.44/task
  • Both harnesses score all 21 tasks with no failures. Quality is identical (0.05 point difference); Strands costs 45% less from prompt caching on Bedrock.

Headline numbers

Harness Mean score Mean $/task Total est. cost Failures
strands 81.21 $2.46 ~$51.70 0
omp 81.16 $4.44 ~$93.24 0

What is included

86 files across two commits:

  • benchmarks/swe-benchmark-data/claude-opus-5-5/strands/swe3/mcp-gateway-registry-v2/ (43 files)
  • benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/mcp-gateway-registry-v2/ (43 files)

Each commit adds eval.json + metrics.json for all 21 tasks, plus the aggregated run-summary.json.

Test plan

  • All 21 tasks produced 6/6 artifacts (both harnesses)
  • All 42 task evaluations scored by the codex judge (0 failures)
  • run-summary.json written by summarize_run.py for both runs
  • No secrets or PII in committed JSON files

Prateek Sinha added 2 commits October 2, 2026 16:56
21 tasks, all scored, mean 81.21, mean cost $2.46/task.
First benchmark of the Strands Agents harness (aarora79#194) on a full dataset.
21 tasks, all scored, mean 81.16, mean cost $4.44/task.
Direct harness comparison with the strands run (81.21 at $2.46/task):
same quality, strands costs 45% less from prompt caching.
@shekharprateek shekharprateek changed the title benchmark(opus-5-5): Opus 5.5 on Bedrock via strands/swe3, v2 dataset benchmark(opus-5-5): Opus 5.5 on Bedrock via strands and omp, v2 dataset Oct 3, 2026
@aarora79
aarora79 merged commit cc15403 into aarora79:main Oct 4, 2026
5 checks passed
aarora79 added a commit that referenced this pull request Oct 5, 2026
…nt Path 2 runs as metered (#210)

* Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier

Regenerate the omp/swe3 v2 frontier JSON, the cost-quality, bubble and radar
charts (light and dark) and the omp report table over every omp v2 run on
disk. Two runs had landed since the last regeneration: claude-opus-5-5 from
#195 (81.16 at $4.44 a task) and kimi-k3-bedrock from #191 (72.87 at $3.87).

Opus 5.5 joins both the combined and the Bedrock frontiers. It scores higher
than claude-sonnet-5 (76.97 at $4.67) and glm-5.3 (80.11 at $7.04) for less,
so those two leave the frontier. kimi-k3-bedrock is dominated by qwen3.8-27b.

Add both models to swe-router's model-aliases.json.

* Rebuild swe-router's models.json and update the frontier claims

Rebuild vend/swe-router/models.json from the new omp/swe3 frontier (21
models). Rewrite the README and results-index sentences the new frontier
made wrong: claude-opus-5-5 (81.16 at $4.44) now sits behind claude-opus-5 in
place of glm-5.3, and replaces claude-sonnet-5 on the Bedrock frontier. Update
the model counts that describe the chart and models.json.

* Ship claude-opus-5-5 in place of glm-5.3 in swe-router's allow-list

The shipped allow-list is the combined cost/quality frontier, which
test_route enforces. Opus 5.5 (81.16, $4.44) replaced glm-5.3 (80.11, $7.04)
on that frontier. Update the comments that counted self-hosted entries and
explained why claude-sonnet-5 is absent.

* Count a Bedrock model reached through LiteLLM as metered on the charts

plot_cost_quality.py and gen_agent_report.py decided a run was metered only
when its provider was "bedrock". A Path 2 run, a Bedrock model reached through
the LiteLLM proxy, records "endpoint", so kimi-k3-bedrock was drawn and grouped
as self-hosted and its report row showed no cost.

Treat an endpoint run as metered when no throughput sweep prices it and it
reports a positive bill. An audit of all 57 endpoint run summaries finds two
such runs, kimi-k3-bedrock (omp) and qwen.qwen3-coder-30b-a3b-instruct (codex),
both Bedrock-billed; every self-hosted run is unchanged. kiro-cli credits stay
their own basis.

Regenerate the omp/swe3 charts, frontier JSON and report. kimi-k3-bedrock
(72.87 at $3.87) joins the Bedrock frontier and claude-opus-4-5 (66.32 at
$4.18) leaves it. The combined frontier is unchanged.

* Rebuild models.json and describe the new Bedrock frontier

Rebuild swe-router's models.json after kimi-k3-bedrock moved to the metered
basis. In the results index, list kimi-k3-bedrock on the Bedrock frontier in
place of claude-opus-4-5, and say Path 2 now has a published run.
aarora79 added a commit that referenced this pull request Oct 10, 2026
omp's --model is a fuzzy match. A Bedrock id missing from the catalog omp
has loaded runs on the nearest model it knows, and omp reports no error.
omp 18.1.15 had no Claude 5.5 entries, so a 21-task "Haiku 5.5" run on
2026-10-08 was Haiku 4.5 on every turn. Upgrading omp is not enough: in a
fresh agent dir, omp -p resolves against the snapshot bundled in the
binary, and 18.8.8 still sends Haiku 5.5 to Haiku 4.5 until `omp models`
has fetched and cached the full catalog in that dir.

run-swe-headless.py now:
- exits before the first task when `omp models amazon-bedrock --json`
  does not list the exact Bedrock id, and says to run `omp update`;
- runs the same listing in each Bedrock task's own agent dir, which
  warms the catalog that omp -p then reads;
- compares the model on every assistant message with the one requested,
  fails a mismatch as the non-retryable `error_model_mismatch`, and
  records the ids in metrics.json as `models_used`.

run-swe-router-headless.py applies the per-dir warm-up and the stream
check to each omp judgment, and raises on a mismatch.

A second bug surfaced in the same run. omp can end a finished session on
stopReason "toolUse"; the harness counted that as an error, retried, and
the retry deleted all six finished artifacts before producing nothing.
A task with every artifact on disk is no longer retried.

The published Opus 5.5 omp row (#195) is unaffected: a hello-world re-run
on omp 18.8.8 answered as us.anthropic.claude-opus-5-5 on every turn, and
its omp-reported cost matches Opus 5.5 rates to the cent, the same rates
#195's $93.27 matches. A fallback to Opus 5 would have cost $157 to $173.
aarora79 added a commit that referenced this pull request Oct 11, 2026
claude-opus-5-5/omp/swe3/Hello-World: the one-task check that settled
whether the published Opus 5.5 omp row ran Opus 5.5. On omp 18.8.8 every
turn named us.anthropic.claude-opus-5-5 (models_used), it scored 91.6,
and its omp-reported $1.1855 matches Opus 5.5 rates to the cent, the same
rates #195's $93.27 matches. Hello-World runs commit their full artifacts
by design; the raw event stream stays ignored.

Two runs go under do-not-include/, which every generator's one-level scan
skips, with a README section each:

- claude-haiku-5-5-resolved-as-haiku-4-5: the 2026-10-08 run requested as
  Haiku 5.5 that omp 18.1.15 fuzzy-matched to Haiku 4.5 on every turn
  (21 generated, 13 judged, 55.58). The evidence behind #218.
- claude-haiku-5-5-misplaced-artifacts: the six artifacts Haiku 5.5 wrote
  for registration-admission-control-gate under the v1 folder instead of
  the v2 path it was given. Never judged; the task stays a failure in the
  published run. Added with -f, since .gitignore keeps run markdown out by
  default. One hunk header in its patch quoted an example token from
  upstream's terraform.tfvars.example; it is redacted. The pre-commit
  hook also stripped trailing whitespace from 25 of its lines; the patch
  still applies at v1.0.19.

The charts and models.json still see 23 models; Haiku 5.5 stays at 72.69.
aarora79 added a commit that referenced this pull request Oct 11, 2026
… v2 dataset (#219)

* benchmark(haiku-5-5, sonnet-5-5): Claude 5.5 on Bedrock via omp/swe3, v2 dataset

Both ran on omp 18.8.8 with the model checks from #218; every assistant
message in every task names the requested model (models_used).

  claude-sonnet-5-5  79.49  $2.62/task  21/21 scored
  claude-haiku-5-5   72.69  $0.14/task  20/21 scored

Haiku 5.5 failed registration-admission-control-gate: it wrote all six
artifacts under mcp-gateway-registry/ instead of the
mcp-gateway-registry-v2/ path the prompt named 56 times, and its retry
wrote nothing. It counts as a failed task, as for any other model.

Both join the combined and the Bedrock frontiers. On the combined
frontier they displace kimi-k3, gemma-4-31b and qwen3.6-35b; on the
Bedrock frontier, claude-haiku-4-5 and kimi-k3-bedrock.

Add Claude Haiku 5.5 to bedrock_pricing.py from the AWS price list
(AmazonBedrockFoundationModels, us-east-1): $0.11 input, $0.55 output,
$0.011 cache read, $0.1375 cache write per million tokens, regional. omp
prices runs from its own catalog, which carries the same rates.

Regenerate the omp/swe3 frontier JSON, the cost-quality, radar and bubble
charts (light and dark) and the omp report. Update the README and
results-index frontier claims, and correct the README badge from "45
models" (a June count of models reachable on Bedrock) to the 28 models
with committed benchmark results, from 11 providers.

* Rebuild swe-router for the Claude 5.5 frontier

Rebuild vend/swe-router/models.json from the new omp/swe3 frontier: 23
models, measured 2026-10-11, source commit 7e8657f.

The shipped allow-list is the combined frontier, which test_route
enforces. It becomes claude-opus-5, claude-opus-5-5, claude-sonnet-5-5,
qwen3.8-27b and claude-haiku-5-5. kimi-k3, gemma-4-31b and qwen3.6-35b
leave it and move to the commented suggestions for teams that self-host.
Only one of the five is now self-hosted, so the file's warning says so.

Add aliases for claude-sonnet-5-5 and claude-haiku-5-5. Regenerate the
README's four per-tier tables from models.json; they had listed 18 models
and now list all 23, and every existing row is unchanged. Add both models
to SKILL.md's tier comparison: Haiku 5.5 scores 79.2 on low and 60.0 on
high, with one hard task unfinished.

Update the route tests that encode the frontier: low work at a floor of
75 now goes to claude-haiku-5-5, high work at 75 still rejects both it
and qwen3.8-27b.

* Keep the Opus 5.5 hello-world check and the two excluded Haiku runs

claude-opus-5-5/omp/swe3/Hello-World: the one-task check that settled
whether the published Opus 5.5 omp row ran Opus 5.5. On omp 18.8.8 every
turn named us.anthropic.claude-opus-5-5 (models_used), it scored 91.6,
and its omp-reported $1.1855 matches Opus 5.5 rates to the cent, the same
rates #195's $93.27 matches. Hello-World runs commit their full artifacts
by design; the raw event stream stays ignored.

Two runs go under do-not-include/, which every generator's one-level scan
skips, with a README section each:

- claude-haiku-5-5-resolved-as-haiku-4-5: the 2026-10-08 run requested as
  Haiku 5.5 that omp 18.1.15 fuzzy-matched to Haiku 4.5 on every turn
  (21 generated, 13 judged, 55.58). The evidence behind #218.
- claude-haiku-5-5-misplaced-artifacts: the six artifacts Haiku 5.5 wrote
  for registration-admission-control-gate under the v1 folder instead of
  the v2 path it was given. Never judged; the task stays a failure in the
  published run. Added with -f, since .gitignore keeps run markdown out by
  default. One hunk header in its patch quoted an example token from
  upstream's terraform.tfvars.example; it is redacted. The pre-commit
  hook also stripped trailing whitespace from 25 of its lines; the patch
  still applies at v1.0.19.

The charts and models.json still see 23 models; Haiku 5.5 stays at 72.69.

* Fix a flaky stray-file sweep test on coarse filesystem clocks

test_untracked_file_written_during_the_level_is_quarantined failed
intermittently on GitHub runners (run 38108751467): it records
since = time.time() and then writes a file, but the kernel stamps mtime
from a coarser clock, so the file can carry an mtime a few milliseconds
before since. _sweep_stray_root_writes skipped any file with
st_mtime < since, so the stray was left in place.

The same rounding would let a real sweep miss a stray written in the
first milliseconds of a level, so fix the check, not the test: accept
mtimes up to _MTIME_SLACK_SECONDS (2.0) before the level start. This
cannot widen what is moved, because a candidate must also be absent from
the snapshot taken at the level start, a plain file, and untracked.

Add a test that backdates a new file 50 ms before since; it fails with
no slack and passes with it.
aarora79 added a commit that referenced this pull request Oct 17, 2026
… dataset (#229)

Claude Opus 5.5 (us.anthropic.claude-opus-5-5) under --agent
strands-harness, the Strands team's create_harness() agent, on the 21
mcp-gateway-registry-v2 tasks. Judged by openai.gpt-5.6-sol at high
effort. All 21 tasks produced six artifacts on the first attempt, and
the loop guard never fired.

  strands-harness  80.80  $2.42/task  21/21 scored
  strands          81.21  $2.46/task  21/21 scored  (#195)
  omp              81.16  $4.44/task  21/21 scored  (#195)

The three harnesses land within 0.41 points of each other, and both
Strands builds cost about 55% of omp per task.

Regenerate harness-strands-harness-swe3.md with gen_agent_report.py and
its frontier JSON and three charts, and restore the hand-written notes
with an Opus 5.5 row. Add the row to docs/strands-harness.md, and fill
in Sonnet 5.5's omp result (79.49, $2.62, from #219), which both pages
still listed as not run.

The combined chart covers Claude Code and pi on v1, and models.json
comes from the omp frontier, so neither changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants